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BACKGROUND 

This brief is in furtherance of the Notice of Appeal, filed in this case on November 

3, 2006. 

The fees required under 37. C:F.R. § 41.20(b)(2), and any required petition for 
extension of time for filing this brief and fees therefore, are dealt with in the accompanying 
TRANSMITTAL OF APPEAL BRIEF. 

This brief comprises these subjects under the headings, and in the order, set forth 

below: 
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The final page of this brief bears the practitioner's signature. 

REAL PARTY IN INTEREST 

The real party in interest in this appeal is Sharp Laboratories of America, Inc., 
assignee of the captioned application. 

RELATED APPEALS AND INTERFERENCES 

There are no other appeals or interferences that will directly affect, be directly 
affected by, or have a bearing on the Board's decision in this appeal. 
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STATUS OF CLAIMS 

A. TOTAL NUMBER OF CLAIMS IN THE APPLICATION 
There are 29 claims currently pending in the application. 

B. STATUS OF ALL CLAIMS 

Claims canceled: 9 
Claims withdrawn: 31-117 
Claims pending: 1-8, 10-30 
Claims allowed: None 
Claims objected to: None 
Claims rejected: 1-8, 10-30. 

C. CLAIMS ON APPEAL 

Claims 1-8 and 10-30 are on appeal. 

A copy of the claims on appeal is set forth in the Claims Appendix to this Brief. 

STATUS OF AMENDMENTS 

No amendment was filed after final rejection. 



SUMMARY OF CLAIMED SUBJECT MATTER 

The claimed subject matter is generally directed to a method of associating additional 
information with a video that includes a plurality of frames. Specifically, in a first embodiment 
exemplified by independent claim 1 , a claimed method may comprise the steps of (a) identifying 
at least one of the frames representative of a picture composed of a plurality of pixels (See FIG. 1 
and Specification at p. 5 line 25 to p. 6 line 1 1); (b) providing a descriptive stream separate from 
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the video (See, e.g. Specification at p. 6 lines 5 line 25 to p. 6 line 1); (c) including the additional 
information in said descriptive stream related to the at least one of said frames (See, e.g. 
Specification at p. 6 lines 6-1 1); (d) providing the video for displaying on a display (See FIG. 3 
reference numeral 84); and (e) selectively providing said additional information to a viewer 
approximately the time of said providing said video where said additional information is an 
object depicted by said picture by said pixels (See, e.g. Specification at p. 26 line 4 to p. 27 line 
22); wherein (f) the additional information includes executable computer program code. (See, 
e.g. Specification at p. 30 lines 11-13). 

In a second embodiment, as exemplified by independent claim 21, a video system may 
comprise: (a) an encoder that includes additional information within a video stream including a 
video including a plurality of frames representative of a picture composed of a plurality of pixels, 
(See, e.g. Specification at p. 13 lines 5-21) where the additional information is related to at least 
one of the frames (See, e.g. Specification at p. 6 lines 6-1 1); (b) a receiver that receives the video 
and the additional information (See Specification at p. 13 lines 18-21), and the receiver decodes 
the video in the same manner independently of whether the additional information is provided 
(See Specification at p. 13 line 22 to p. 14 line 16); (c) a display for displaying the video (See 
FIG. 3, reference numeral 84); (d) a trigger mechanism for selectively presenting the additional 
information to a viewer at approximately the time of presenting the frames to the viewer where 
the additional information is an object depicted by the picture by the pixels (See, e.g. 
Specification at p. 26 line 4 to p. 27 line 22); wherein (e) the additional information includes 
executable computer program code (See, e.g. Specification at p. 30 lines 1 1-13). 
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GROUNDS OF REJECTION TO BE REVIEWED ON APPEAL 



The grounds of rejection presented for review are: (1) whether claims 1-8, 10-12, and 15- 
30 are unpatentable under 35 U.S.C. § 102(b) as being anticipated over Wistendahl et al, U.S. 
Patent No. 5,708, 845 (hereinafter Wistendahl); and (2) whether claims 13 and 14 are 
unpatentable under 35 U.S.C. § 103(a) in view of Wistendahl. 

ARGUMENT 

REJECTION UNDER 35 U.S.C. §102(b) OVER WISTENDAHL 

The Examiner rejected claims 1-8, 10-12 and 15-30 under 35 U.S.C. § 102(b) as being 
anticipated by Wistendahl. Wistendahl discloses a system for viewing an interactive digital 
media (IDM) presentation where user selection of u hot spots" in a video display is intended to 
trigger presentation of additional material to the viewer. Specifically, to facilitate platform 
independent viewing of the IDM presentations, the cited reference discloses a data layer ("N- 
data") that maps those hot spots to a coordinate system of the display device, so as to avoid 
embedding any platform-dependent codes in the frame data of the IDM presentation itself. The 
"N-data" layer is transparent to the user, i.e. the user is not aware of it. For example, the IDM 
presentation may include a "hot spot" comprising an object which, if selected, triggers a 
hyperlink to additional content, the hyperlink being embedded in program running the IDM 
presentation. The N-data maps this hot spot to a coordinate system, such that, if a user activates 
those coordinates, the hyperlink is activated. 

An example recited in the cited reference is of viewer watching The Maltese Falcon in an 
IDM presentation, where an N-data layer maps hot spots that, when selected, trigger pop-up 
movie trivia about the stars, e.g. Humphrey Bogart, etc. 



Independent claims 1 and 21 include the limitations of (1) "providing a descriptive 
scheme" that is "separate from [the] video [frames]' 5 and that includes "additional information . . 
. related to ... at least one of the frames" and (2) "providing said additional information to a 
viewer" where "said additional information is an object depicted by said picture" and "includes 
executable computer code." Wistendahl fails to disclose these latter limitations. The Examiner 
argues that it is the N-data layer and the IDM program code comprise the "additional 
information." The Examiner then contends that Wistendahl 5 s "additional information 55 includes 
"executable computer code" in the form of "hyperlink and trigger information used to launch 
web sites and/or applications." The flaw in the Examiner's rejection, however, is that neither the 
N-data nor the IDM program code are objects depicted by said picture. Though the N-data maps 
objects in the pictures to a coordinate system, the N-data cannot be said to be the objects 
themselves that are mapped. Moreover, the IDM program code merely implements instructions 
that are to be carried out when N-data coordinates are selected. 

Therefore, each of independent claims 1 and 21, as well as their respective dependent 
claims 2-8, 10-12, 15-20, and 22-30, patentably distinguish over the cited prior art, and the 
Examiner's rejection of these claims should be overturned. 

REJECTION UNDER 35 U.S.C. § 103(a) OVER WISTENDAHL 

Dependent claims 13 and 14 respectively depend from independent claim 1 and are 
patentably distinguished over Wistendahl fro the same reasons as is independent claim 1 . 
Therefore, the Examiner's rejection of claims 13 and 14 should also be overturned. 
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CONCLUSION 

The Examiner's respective rejections of claims 1-8, and 10-30 should be reversed, and 
the claims should be found patentable. 



Respectfully submitted, 




Kurt Rohlfs 
Reg. No. 54,405 
Attorney for Applicant 
Telephone: (503) 227-5631 
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CLAIMS APPENDIX 



1 . A method of associating additional information with a video including a plurality of 
frames comprising: 

(a) identifying at least one of said frames representative of a picture 
composed of a plurality of pixels; 

(b) providing a descriptive stream separate from said video; 

(c) including said additional information in said descriptive stream related to 
said at least one of said frames; 

(d) providing said video for displaying on a display; 

(e) selectively providing said additional information to a viewer 
approximately the time of said providing said video where said additional 
information is an object depicted by said picture by said pixels; 

(f) wherein said additional information includes executable computer 
program code. 

2. The method of claim 1 wherein said additional information includes at least one 
of an object index, a textual description, a voice annotation, an image feature, an object link, a 
URL link, and a Java applet. 

3. The method of claim 1 wherein said identifying is an object within said frame. 

4. The method of claim 1 where said descriptive stream is related to a plurality of 
said frames. 

5. The method of claim 4 wherein said at least one of said frames are in sequential 
order in said video. 

6. The method of claim 4 wherein said at least one of said frames are in 
nonsequential order in said video. 
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7. A method of claim 3 wherein said additional information is related to said object. 

8. The method of claim 1 wherein said descriptive stream includes an index 
synchronizing said video with said descriptive stream. 

9 (Canceled). 

10. The method of claim 1 wherein said descriptive stream is encoded separately from 
said video. 

11. The method of claim 10 wherein said video is decoded in the same manner 
independently of whether said descriptive stream is provided. 

12. The method of claim 1 1 wherein said video is at least one of MPEG-2 and 
television broadcast format. 

13. The method of claim 1 wherein said additional information is presented to said 
viewer on a remote control. 

14. The method of claim 1 wherein an audible signal indicates the availability of said 
additional information. 

15. The method of claim 1 wherein a visual signal indicates the availability of said 
additional information. 

16. The method of claim 7 wherein said additional information includes textual based 
information related to said object. 

17. The method of claim 7 wherein said additional information includes audible 
information related to said object. 
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18. The method of claim 7 wherein said additional information includes image 
features comprising at least one of texture, shape, dominant color, and a motion model related to 
said object. 

19. The method of claim 7 wherein said additional information includes links to at 
least one of other objects and frames within said video. 

20. The method of claim 7 wherein said additional information includes program 
instructions related to said object. 

21. A video system comprising: 

(a) 



(b) 

(c) 
(d) 



(e) 

22. 

(a) 
(b) 
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an encoder that includes additional information within a video stream 
including a video including a plurality of frames representative of a 
picture composed of a plurality of pixels, where said additional 
information is related to at least one of said frames; 
a receiver that receives said video and said additional information, and 
said receiver decodes said video in the same manner independently of 
whether said additional information is provided; 
a display for displaying said video; 

a trigger mechanism for selectively presenting said additional information 
to a viewer at approximately the time of presenting said frames to said 
viewer where said additional information is an object depicted by said 
picture by said pixels; 

wherein said additional information includes executable computer 
program code. 

The system of claim 21, further comprising: 
a transmitter for transmitting said video signal and said additional 
information; and 

a receiver for receiving said yideo signal and said additional information. 



23. 

a computer. 



The system of claim 22 wherein said encoder is at least one of a video camera and 



24. The system of claim 21 wherein said trigger mechanism is located in a remote 
control device. 

25. The system of claim 21 wherein said additional information is provided by a 
remote control device. 

26. The method of claim 21 wherein said additional information is related to an object 
within said frame and includes links to at least one of other objects and frames within said video. 

27. The method of claim 21 wherein said additional information is related to an object 
within said frame and includes program instructions related to said object. 

28. The method of claim 21 wherein said additional information is related to an object 
within said frame and includes textual based information related to said object. 

29. The method of claim 21 wherein said additional information is related to an object 
within said frame and includes audible information related to said object. 

30. The method of claim 21 wherein said additional information is related to an object 
within said frame and includes image features comprising at least one of texture, shape, 
dominant color, and a motion model related to said object. 

31 (Withdrawn). A system for presenting information comprising: 

(a) a single unitary file stored on a recordable media containing both only a 
single image representative of a picture composed of a plurality of pixels 
and additional information associated with said image, wherein said image 
is stored in a first part of said file and said additional information is stored 
in a second part of said file, said single unitary file does not contain any 
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other images than said single image, wherein said single unity file is not 
part of a video sequence; 

(b) a selection mechanism that reads said single image from said single unity 
file and permits the selection of graphical objects depicted by said picture 
by said pixels in said image for which said additional information is 
related thereto; and 

(c) a presentation mechanism that provides said additional information to a 
viewer in response to selecting said object depicted by said picture by said 
pixels. 

32 (Withdrawn). The system of claim 3 1 wherein said file includes said image 
followed by said additional information. 

33 (Withdrawn). The system of claim 32 wherein said image and said additional 
information are separated by a marker indicating the end of said image. 

34 (Withdrawn). The system of claim 33 wherein an image viewer which does not 
recognize said additional information will display said image properly and recognize said marker 
as indicating the end of said image. 

35 (Withdrawn). The system of claim 34 wherein said image is in a JPEG format. 

36 (Withdrawn). The system of claim 31 wherein said additional information is 
organized in at least two layers comprising: 

(a) a first layer containing information describing the location of objects 
within said image; and 

(b) a second layer containing additional information regarding said objects 
within said image, where said first layer contains fewer bytes than said 
second layer. 
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37 (Withdrawn). The system of claim 36 wherein said second layer follows said first 
layer, which in turn follows said image file. 

38 (Withdrawn). The system of claim 36 wherein said first layer contains a length 
identifier describing the length of said first layer. 

39 (Withdrawn). * The system of claim 36 wherein said first layer contains a number 
of objects identifier describing the number of objects identified by said first layer. 

40 (Withdrawn). The system of claim 36 wherein said first layer contains a number 
of data identifier describing the number of data items associated with a particular said object. 

41 (Withdrawn). The system of claim 36 wherein said first layer contains a first 
definition of the outline of an object of said image. 

42 (Withdrawn). The system of claim 36 wherein said second layer contains a 
length identifier describing the length of said second layer. 

43 (Withdrawn). The system of claim 36 wherein said second layer contains an 
array of offsets that identify the start of each data item. 

44 (Withdrawn). The system of claim 41 wherein said second layer contains a 
second definition of the outline of said object of said image, where said second definition more 
closely approximates the outline of said object than said first definition. 

45 (Withdrawn). The system of claim 41 wherein said second layer contains a 
second definition of the outline of said object of said image, where said second definition 
contains more bytes than said first definition. 

46 (Withdrawn). The system of claim 36 wherein said second layer includes sound 
data related to said object. 
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47 (Withdrawn). The system of claim 36 wherein said second layer includes HTML 
meta tags related to said object. 

48 (Withdrawn). The system of claim 36 wherein said second layer includes textual 
annotations related to said object. 

49 (Withdrawn). The system of claim 36 wherein said second layer includes an 
HTML page to be rendered. 

50 (Withdrawn). The system of claim 36 wherein said second layer includes a Java 
Applet related to said object. 

5 1 (Withdrawn). The system of claim 36 wherein said second layer includes a color 
histogram. 

52 (Withdrawn). The system of claim 36 wherein said second layer includes data 
related to the conditions under which said image was created including at least one of lighting, 
camera settings, and time of acquisition. 

53 (Withdrawn). The system of claim 36 wherein said second layer includes data 
related to information for reproducing said image including at least one of cropping information, 
paper type, camera settings, and image production settings. 

54 (Withdrawn). The system of claim 36 wherein said second layer includes another 
image to be superimposed upon said image. 

55 (Withdrawn). The system of claim 36 wherein said second layer includes data 
regarding the author of said image. 
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56 (Withdrawn). The system of claim 36 wherein said second layer includes 
copyright data regarding the copyright of said image. 

57 (Withdrawn). The system of claim 56 said copyright data is encoded. 

58 (Withdrawn). The system of claim 36 wherein said second layer includes 
information regarding how said image should be viewed. 

59 (Withdrawn). A method of presenting information comprising: 

(a) displaying an image from a single unitary file stored on a recordable 
media_containing both only a single image representative of a picture 
composed of a plurality of pixels and additional information associated 
with said image, where said image is stored in a first part of said file and 
said additional information is stored in a second part of said file, said 
single unitary file does not contain any other images than said single 
image, wherein said single unity file is not part of a video sequence; 

(b) selecting an object depicted by said picture by said pixels in said image for 
which said additional information is related thereto; and 

(c) providing said additional information to a viewer in response to selecting 
said object depicted by said picture by said pixels. 

60 (Withdrawn). The method of claim 59 wherein said file includes said image 
followed by said additional information. 

61 (Withdrawn). The method of claim 60 wherein said image and said additional 
information are separated by a marker indicating the end of said image. 

62 (Withdrawn). The method of claim 61 wherein an image viewer which does not 
recognize said additional information will display said image properly and recognize said marker 
as indicating the end of said image. 

63 (Withdrawn). The method of claim 62 wherein said image is in a JPEG format. 
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64 (Withdrawn). The method of claim 59 wherein said additional information is 
organized in at least two layers comprising a first layer containing information describing the 
location of objects within said image and a second layer containing additional information 
regarding said objects within said image, where said first layer contains fewer bytes than said 
second layer. 

65 (Withdrawn). The method of claim 64 wherein said second layer follows said 
first layer, which in turn follows said image file. 

66 (Withdrawn). The method of claim 64 wherein said first layer contains a length 
identifier describing the length of said first layer. 

67 (Withdrawn). The method of claim 64 wherein said first layer contains a number 
of objects identifier describing the number of objects identified by said first layer. 

68 (Withdrawn). The method of claim 64 wherein said first layer contains a number 
of data identifier describing the number of data items associated with a particular said object. 

69 (Withdrawn). The method of claim 64 wherein said first layer contains a first 
definition of the outline of an object of said image. 

70 (Withdrawn). The method of claim 64 wherein said second layer contains a 
length identifier describing the length of said second layer. 

71 (Withdrawn). The method of claim 64 wherein said second layer contains an 
array of offsets that identify the start of each data item. 

72 (Withdrawn). The method of claim 69 wherein said second layer contains a 
second definition of the outline of said object of said image, where said second definition more 
closely approximates the outline of said object than said first definition. 
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73 (Withdrawn). The method of claim 69 wherein said second layer contains a 
second definition of the outline of said object of said image, where said second definition 
contains more bytes than said first definition. 

74 (Withdrawn). The method of claim 64 wherein said second layer includes sound 
data related to said object. 

75 (Withdrawn). The method of claim 64 wherein said second layer includes HTML 
meta tags related to said object. 

76 (Withdrawn). The method of claim 64 wherein said second layer includes textual 
annotations related to said object. 

77 (Withdrawn). The method of claim 64 wherein said second layer includes an 
HTML page to be rendered. 

78 (Withdrawn). The method of claim 64 wherein said second layer includes a Java 
Applet related to said object. 

79 (Withdrawn). The method of claim 64 wherein said second layer includes a color 
histogram. 

80 (Withdrawn). The method of claim 64 wherein said second layer includes data 
related to the conditions under which said image was created including at least one of lighting, 
camera settings, and time of acquisition. 

8 1 (Withdrawn). The method of claim 64 wherein said second layer includes data 
related to information for reproducing said image including at least one of cropping information, 
paper type, camera settings, and image production settings. 
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82 (Withdrawn). The method of claim 64 wherein said second layer includes 
another image to be superimposed upon said image. 

83 (Withdrawn). The method of claim 64 wherein said second layer includes data 
regarding the author of said image. 

84 (Withdrawn). The method of claim 64 wherein said second layer includes 
copyright data regarding the copyright of said image. 

85 (Withdrawn). The method of claim 64 said copyright data is encoded. 

86 (Withdrawn). The method of claim 64 wherein said second layer includes 
information regarding how said image should be viewed. 

87 (Withdrawn). A method for creating an image file comprising: 

(a) providing a single image representative of a picture composed of a 
plurality of pixels; 

(b) selecting an object depicted by said picture by said pixels in said image; 

(c) providing additional information related to said object; and 

(d) storing both said single image and said additional information in a single 
unitary file stored on a recordable media, where said file includes only a 
single image and said image is stored in a first part of said file and said 
additional information is stored in a second part of said file, said single 
unitary file does not contain any other images than said single image, 
wherein said single unity file is not part of a video sequence. 

88 (Withdrawn). The method of claim 87 wherein said file includes said image 
followed by said additional information. 

89 (Withdrawn). The method of claim 88 wherein said image and said additional 
information are separated by a marker indicating the end of said image. 
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90 (Withdrawn). The method of claim 89 wherein an image viewer which does not 
recognize said additional information will display said image properly and recognize said marker 
as indicating the end of said image. 

91 (Withdrawn). The method of claim 90 wherein said image is in a JPEG format. 

92 (Withdrawn). The method of claim 87 wherein said additional information is 
organized in at least two layers comprising a first layer containing information describing the 
location of objects within said image and a second layer containing additional information 
regarding said objects within said image, where said first layer contains fewer bytes than said 
second layer. 

93 (Withdrawn). The method of claim 92 wherein said second layer follows said 
first layer, which in turn follows said image file. 

94 (Withdrawn). The method of claim 92 wherein said first layer contains a length 
identifier describing the length of said first layer. 

95 (Withdrawn). The method of claim 92 wherein said first layer contains a number 
of objects identifier describing the number of objects identified by said first layer. 

96 (Withdrawn). The method of claim 92 wherein said first layer contains a number 
of data identifier describing the number of data items associated with a particular said object. 

97 (Withdrawn). The method of claim 92 wherein said first layer contains a first 
definition of the outline of an object of said image. 

98 (Withdrawn). The method of claim 92 wherein said second layer contains a 
length identifier describing the length of said second layer. 
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99 (Withdrawn). The method of claim 92 wherein said second layer contains an 
array of offsets that identify the start of each data item. 

100 (Withdrawn). The method of claim 97 wherein said second layer contains a 
second definition of the outline of said object of said image, where said second definition more 
closely approximates the outline of said object than said first definition. 

101 (Withdrawn). The method of claim 97 wherein said second layer contains a 
second definition of the outline of said object of said image, where said second definition 
contains more bytes than said first definition. 

102 (Withdrawn). The method of claim 92 wherein said second layer includes sound 
data related to said object. 

103 (Withdrawn). The method of claim 92 wherein said second layer includes HTML 
meta tags related to said object. 

104 (Withdrawn). The method of claim 92 wherein said second layer includes textual 
annotations related to said object. 

105 (Withdrawn). The method of claim 92 wherein said second layer includes an 
HTML page to be rendered. 

106 (Withdrawn). The method of claim 92 wherein said second layer includes a Java 
Applet related to said object. 

107 (Withdrawn). The method of claim 92 wherein said second layer includes a color 
histogram. 
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108 (Withdrawn). The method of claim 92 wherein said second layer includes data 
related to the conditions under which said image was created including at least one of lighting, 
camera settings, and time of acquisition. 

109 (Withdrawn). The method of claim 92 wherein said second layer includes data 
related to information for reproducing said image including at least one of cropping information, 
paper type, camera settings, and image production settings. 

1 10 (Withdrawn). The method of claim 92 wherein said second layer includes 
another image to be superimposed upon said image. 

1 1 1 (Withdrawn). The method of claim 92 wherein said second layer includes data 
regarding the author of said image. 

1 12 (Withdrawn). The method of claim 92 wherein said second layer includes 
copyright data regarding the copyright of said image. 

1 13 (Withdrawn). The method of claim 92 said copyright data is encoded. 

1 14 (Withdrawn). The method of claim 92 wherein said second layer includes 
information regarding how said image should be viewed. 

1 1 5 (Withdrawn). The method of claim 92 wherein said first layer is transmitted from 
a first computer to a second computer together with said image. 

1 16 (Withdrawn). The method of claim 1 15 wherein portions of said second layer are 
transmitted from said first computer to said second computer upon request by said first 
computer. 

117 (Withdrawn). The method of claim 1 16 wherein said request is in response to a 
user selecting an object within said image. 
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EVIDENCE APPENDIX; 

None. 



RELATED PROCEEDINGS APPENDIX: 

None. 
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